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Social  interactions  to  supplement  learning  and  asynchronous  tools  to  facilitate  exchange  of  quality  ideas 
have  gained  much  attention  in  information  systems  education.  While  various  systems  exist,  students 
have  difficulty  with  deep  processing  of  complex  instructional  materials  (e.g.,  concepts  of  a  theory  and 
pedagogical  support  mechanisms  derived  from  a  theory).  This  research  proposes  a  theoretical  framework 
that  leverages  attention  guidance  in  a  social  constructivist  approach  to  facilitate  processing  of  central 
domain  concepts,  principles,  and  their  interrelations.  Using  an  open  source  anchored  discussion  system, 
we  designed  a  set  of  instructor-based  and  peer-oriented  attention  guidance  functionalities  involving 
dynamic  manipulation  of  text  font  size  similar  to  tag  clouds.  We  conducted  an  experimental  study  with 
two  small  groups  of  first-year  doctoral  students  in  a  blended-learning  classroom  format.  Students  in  the 
control  group  had  no  access  to  attention  guidance  functions.  Students  in  the  treatment  group  used 
instructor-based  attention  guidance  functionality  and  then  switched  to  peer-oriented  attention  guidance 
functionality.  The  evaluation  compared  focus,  content,  and  sequential  organization  of  students’  online 
discussion  messages  with  heat  maps,  content  analysis,  sequential  analysis,  and  statistical  discourse 
analysis  to  examine  different  facets  of  the  phenomenon  in  a  holistic  way.  The  results  show  that  in  areas 
where  students  struggle  to  understand  challenging  concepts,  instructor-based  attention  guidance 
functionality  facilitated  elaboration  and  negotiation  of  ideas,  which  is  fundamental  to  higher  order 
thinking.  In  addition,  after  switching  to  peer-oriented  attention  guidance  functionality,  students  in  the 
treatment  group  took  the  lead  in  pinpointing  challenging  concepts  they  did  not  previously  understand. 
These  findings  indicate  that  instructor-based  and  peer-oriented  attention  guidance  functionalities  offer 
students  an  indirect  way  of  focusing  their  attention  on  deep  processing  of  challenging  concepts  in  an 
inherently  open  learning  environment.  Implications  for  theory,  software  design,  and  future  research  are 
discussed. 

Published  by  Elsevier  Ltd. 


1.  Introduction 

Collaborative  learning  (CL)  opens  opportunities  for  an  individual  to  actively  construct  new  knowledge  for  a  deep  understanding  of  a 
subject  matter.  A  key  element  in  the  effectiveness  of  this  popular  educational  approach  is  social  interaction  (Phielix,  Prins,  Kirschner,  Erkens, 
&  Jaspers,  2011 ).  Social  interaction  can  provide  a  natural  setting  for  demanding  cognitive  activities  such  as  externalizing  one’s  own 
perspective,  questioning  collaborators’  perspectives,  negotiating  differences  in  perspectives,  and  internalizing  a  refined  perspective  (Dehler, 
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Bodemer,  Buder,  &  Hesse,  2011).  These  activities  allow  students  to  use  one  another  as  a  resource  for  collaborative  learning.  Thus,  the 
effectiveness  of  collaborative  learning  relies  on  the  quality  of  social  interaction. 

Computer-supported  collaborative  learning  (CSCL)  systems  can  facilitate  and  stimulate  the  production  of  high  quality  social  interactions 
with  their  functional  characteristics  (Suthers,  2006).  An  annotation-based  asynchronous  online  discussion  system  is  one  valuable  CSCL  tool 
for  collaborative  processing  of  academic  texts.  This  system  is  sometimes  referred  to  as  anchored  discussion  because  students’  shared  an¬ 
notations  directly  link  discussion  threads  to  relevant  context,  affording  greater  coherence  (Brush,  Bargeron,  Grudin,  Borning,  &  Gupta,  2002; 
Guzdial  &  Turns,  2000;  Van  der  Pol,  Admiraal,  &  Simons,  2006).  There  are  two  linking  functions  crucial  to  the  design  of  an  anchored  dis¬ 
cussion  system:  artifact-to-discussion  linking  and  discussion-to-artifact  linking.  Suthers  (2001 )  reported  that  these  linking  functions  tightly 
couple  instructional  material  (artifact)  and  its  related  discussion.  Extensive  prior  research  compared  anchored  discussion  with  conventional 
threaded  discussion.  For  example,  Guzdial  and  Turns  (2000)  showed  that  linking  a  discussion  thread  to  an  entire  or  a  section  of  an 
instructional  material  stimulates  more  sustained  and  on-topic  social  interaction.  Brush  et  al.  (2002)  obtained  similar  results  and  further 
demonstrated  that  linkage  to  finer-grained  material  (words  or  passages)  promotes  focused  messages.  Next,  Muhlpford  and  Wessner  (2005) 
found  that  these  linking  functions  act  as  a  catalyst  for  building  common  ground  in  social  interaction.  Along  this  line.  Van  der  Pol  et  al.  (2006) 
observed  that  an  anchored  discussion  system  increases  communicative  efficiency,  encourages  re-reading  of  relevant  passages  from  an 
instructional  material,  and  produces  meaning-oriented  discussion.  Finally,  Eryilmaz,  Ryan,  Van  der  Pol,  Kasemvilas,  and  Mary  (2013)  iso¬ 
lated  the  combined  effects  of  two  linking  functions  from  the  presence  of  instructional  material  in  an  anchored  discussion  system  and 
developed  a  theoretical  model  to  explain  the  resulting  quality  and  flow  of  social  interactions. 

Despite  the  positive  results  obtained  by  previous  research,  there  are  still  many  problems  concerning  the  collaborative  processing  of 
academic  texts  with  anchored  discussion  systems.  A  pressing  problem,  as  stressed  by  Peters  and  Hewitt  (2010),  is  that  students  have  dif¬ 
ficulty  with  deep  processing  of  central  domain  concepts,  principles,  and  their  interrelations  from  complex  instructional  materials  (e.g., 
concepts  of  a  theory  and  pedagogical  support  mechanisms  derived  from  a  theory).  The  factors  that  give  rise  to  this  problem  are  twofold. 
First,  students  may  opt  for  sharing  existing  opinions  and  experiences  in  online  discussions  due  to  a  natural  tendency  to  get  things  done  with 
minimal  effort  (Sandberg  &  Barnard,  1997).  In  this  line  of  reasoning,  Hewitt  (2005)  found  that  students  gravitate  to  familiar  (comfortable) 
topics  and  avoid  challenging  topics  from  complex  instructional  materials  in  order  to  meet  online  discussion  participation  requirements. 
According  to  Betrancourt,  Dillenbourg,  and  Clavien  (2008),  students’  above  mentioned  tendencies  induce  a  “shallow  processing  of  the 
subject  matter  instead  of  a  deeper  and  more  demanding  processing”  (p.  66).  Along  this  line,  Van  der  Pol  et  al.  (2006)  referred  to  shallow 
processing  as  “opinion-oriented  discussion”  and  deep  processing  as  “meaning-oriented  discussion”  within  an  anchored  discussion  system. 
Second,  Kim  and  Hannafin  (2011)  found  that  when  students  have  low  levels  of  prior  knowledge,  naive  assumptions  situated  in  prior  ex¬ 
periences  limit  or  fail  to  adequately  inform  their  collaborative  knowledge  construction.  Under  such  conditions,  Kim  and  Hannafin  (2011 ) 
remarked  that  “students  develop  robust  and  oversimplified  misconceptions  that  prove  highly  resilient  to  change"  (p.  412).  Taken 
together,  both  factors  underscore  students’  difficulties  in  productive  use  of  instructional  resources  during  anchored  discussions,  which  can 
yield  lower  learning  results  (Jeong  &  Hmelo-Silver,  2010).  Building  on  this  line  of  reasoning,  we  concur  with  Lin  and  Atkinson  (2011)  that 
students  with  low  levels  of  prior  knowledge  can  benefit  from  attention  guidance  in  collaborative  processing  of  challenging  concepts  from 
complex  instructional  materials.  However,  a  central  challenge  with  attention  guidance  is  that  as  students  become  more  familiar  with  a 
subject  matter,  instructor’s  guidance  may  become  redundant  and  in  some  cases  it  may  even  hinder  collaborative  knowledge  construction 
because  students  may  rely  too  much  on  an  instructor’s  expertise  and  authority  (Puntambekar  &  Hubscher,  2005).  Therefore,  the  notion  of 
successful  scaffolding  entails  the  fading  of  an  instructor’s  guidance,  which  forces  students  to  practice  their  own  knowledge  and  develop  the 
necessary  skills  for  novel  situations  (Wecker  &  Fischer,  2007). 

In  order  to  address  the  problem  at  hand,  we  examined  the  design  and  evaluation  of  two  different  attention  guidance  functionalities  in  an 
open  source  anchored  discussion  system:  instructor-based  and  peer-oriented  attention  guidance.  Our  central  argument  involved  two 
components.  First,  instructor-based  attention  guidance  functionality  would  help  students  with  low  prior  domain  knowledge  to  select  and 
subsequently  process  a  complex  instructional  material’s  challenging  concepts  with  high  quality  social  interaction  patterns.  Second,  students 
would  continue  to  carry  out  high  quality  social  interaction  patterns  focusing  on  challenging  concepts  after  switching  from  instructor-based 
to  peer-oriented  attention  guidance  functionality. 

The  article  proceeds  as  follows.  The  next  section  leverages  attention  guidance  in  a  social  constructivist  perspective  to  facilitate  focused 
online  learning  conversations.  Building  on  this  theoretical  framework,  the  subsequent  section  describes  the  design  of  instructor-based  and 
peer-oriented  attention  guidance  functionalities  in  an  open  source  anchored  discussion  system.  The  sections  following  that  detail  the 
research  hypotheses,  methodological  approach,  and  present  results  of  the  study.  The  article  concludes  with  a  discussion  of  the  findings, 
contributions,  limitations,  as  well  as  avenues  for  future  research. 

2.  Theoretical  framework 

The  theoretical  lens  that  guides  our  study  is  an  “interactional  constructivist  epistemology”  (Suthers,  2006).  This  position  emphasizes  that 
the  value  of  social  interaction  lies  in  “idea  improvement”  (Scardamalia  &  Bereiter,  2006,  p.  99)  within  small  learning  groups.  Two  theoretical 
stances  that  were  introduced  by  Mayer  and  colleagues  (e.g.,  Robins  &  Mayer,  1993)  serve  as  the  underpinning  of  this  position:  active 
responding  and  active  processing.  The  first  stance,  active  responding,  recognizes  the  value  of  making  intellectual  advances  in  areas  where 
students  struggle  to  understand,  such  as  expressing  ideas  to  clarify  misconceptions;  asking  thought-provoking  questions  to  pursue  a  new 
line  of  reasoning;  elaborating  on  existing  ideas  to  extend  what  is  already  known;  and  negotiating  conceptual  discrepancies  to  justify,  defend, 
and  revise  ideas  (Kobbe  et  al.,  2007).  The  second  stance,  active  processing,  classifies  these  types  of  social  interaction  activities  as  constructive 
or  truly  mathemagenic  (Rothkopf,  1970)  only  if  their  outputs  contain  ideas  that  go  beyond  the  explicitly  presented  information  in  a  complex 
instructional  material  (Chi,  2009).  For  example,  asking  students  to  compare-and-construct  the  abstract  knowledge  of  two  complex 
instructional  materials  in  social  interaction  requires  them  to  express  the  relevant  similarities  and  differences  in  their  own  words  when  the 
instructional  materials  do  not  explicitly  state  this  information.  In  summary,  the  term  active  characterizes  the  production  of  such  activities  in 
social  interaction  rather  than  passive  acceptance  of  peers’  inputs,  and  these  actives  become  constructive  when  they  produce  new  ideas  that 
go  beyond  the  information  provided  in  a  complex  instructional  material. 
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However,  learning  from  social  interaction  may  be  hindered  when  students  feel  overwhelmed  with  the  discovery  of  a  new  knowledge 
that  no  one  in  discussion  yet  possesses  and  the  participants  resort  to  shallow  processing  strategies  in  a  computer-supported  collaborative 
learning  (CSCL)  environment,  a  phenomenon  typically  referred  to  as  lost  in  hyperspace  (Scheiter  &  Gerjets,  2007).  Under  such  conditions, 
prior  research  reported  that  online  discussions  drift  from  one  familiar  topic  to  another,  while  leaving  important  threads  to  die  without 
diagnosing  and  resolving  challenging  misconceptions  (Eklundh  &  Rodriguez,  2004;  Hewitt,  2005;  Lipponen,  Rahikainen,  Lallimo,  & 
Hakkarainen,  2003;  Potter,  2008).  As  a  result  of  this  neglect,  these  discussion  threads  read  more  like  reflective  monologues  than  dialog¬ 
ical  interactions  (see  Pena-Shaff  &  Nicholls,  2004  for  a  study  that  involves  both  undergraduate  and  graduate  level  students).  The  crucial 
point  of  this  problem,  as  demonstrated  by  Engelmann  and  Hesse  (2011 ),  is  that  when  misconceptions  remain  undiscovered  students  fail  to 
grasp  important  information.  This  problem  is  especially  important  when  research  on  the  effects  of  collaborative  learning  is  highly  incon¬ 
clusive  (see  Kirschner,  Paas,  &  Kirschner,  2009;  Phielix,  Prins,  &  Kirschner,  2010  for  reviews).  In  pursuing  our  research  on  the  design  and 
evaluation  of  CSCL  systems,  we  build  on  Renkl  and  Atkinson  (2007)  to  argue  that  students’  active  processing  should  not  only  relate  to 
learning  contents,  but  also  focus  on  central  domain  concepts  and  principles  that  impact  learning  more  directly.  For  example,  in  the  subject  of 
constructivist  instruction,  such  concepts  and  principles  may  include  relating  teacher  actions  in  a  constructivist  classroom  to  student 
learning  and  developing  students’  metawareness  of  their  own  understanding  and  learning  processes. 

Attention-guiding  cues  have  the  potential  to  focus  students’  attention  to  challenging  concepts  from  complex  instructional  materials.  We 
define  attention-guiding  cues  as  the  manipulation  of  visuospatial  characteristics  (e.g.,  font  size)  of  text-based  instructional  materials  in 
order  to  capture  students’  attention  in  an  involuntary  or  obligatory  fashion.  This  exogenous  cueing  method  with  origins  in  text-processing 
research  therefore  does  not  add  extra  content  information  or  change  the  existing  content  of  complex  instructional  materials  (De  Koning, 
Tabbers,  Rikers,  &  Paas,  2010a;  Van  Gog,  Jarodzka,  Scheiter,  Gerjets,  &  Paas,  2009).  A  large  body  of  research  has  demonstrated  that  for 
students  with  low  prior  domain  knowledge,  attention-guiding  cues  are  an  effective  means  of  selecting  challenging  concepts  from  complex 
instructional  materials.  To  begin,  Lorch  and  Lorch  (1995)  found  that  visual  cues  slow  the  reading  times  for  text  processing.  In  line  with  this 
finding,  De  Koning,  Tabbers,  Rikers,  and  Paas  (2010b)  showed  that  visual  cues  facilitate  students  to  look  more  often  and  for  longer  periods  of 
time  at  cued  than  non-cued  contents  (for  similar  findings  see  Schnotz  &  Lowe,  2008).  Furthermore,  Irwin,  Colcombe,  Kramer,  and  Hahn 
(2000)  remarked  that  students  look  at  cued  contents  first  even  if  they  are  no  more  important  than  non-cued  contents.  In  terms  of  effi¬ 
ciency,  Kalyuga,  Chandler,  and  Sweller  (1999)  demonstrated  that  visual  cues  promote  efficient  learning  as  they  decrease  perceived  difficulty 
of  finding  challenging  concepts,  which  increases  the  availability  of  group  time  to  collaboratively  process  those  concepts.  Taken  together,  the 
findings  from  these  previous  studies  suggest  that  attention-guiding  cues  offer  students  an  indirect  way  of  focusing  their  social  interaction  on 
the  processing  of  challenging  concepts  from  complex  instructional  materials. 

But,  does  attention  guidance  improve  learning  performance  in  a  systematic  way?  On  the  one  hand,  Boucheix  and  Lowe  (2010)  showed 
that  attention  guidance  supports  construction  of  a  mental  model  of  causal  chains  in  cued  areas.  Similarly,  De  Koning,  Tabbers,  Rikers,  and 
Paas  (2007)  found  that  attention  guidance  improves  retention  and  transfer  performance.  On  the  other  hand,  Kriz  and  Hegarty  (2007)  failed 
to  find  better  learning  outcomes  for  cued  compared  to  non-cued  instructional  resources.  In  other  words,  cues  may  help  students  with 
locating  specific  information  of  an  instructional  material;  however,  cues  do  not  necessarily  help  students  with  comprehension  of  infor¬ 
mation.  As  pointed  by  De  Koning,  Tabbers,  Rikers,  and  Paas  (2011),  one  possible  explanation  for  this  discrepancy  is  that  cues  are  most 
beneficial  for  learning  when  instructional  materials  have  high  complexity  because  quickly  locating  challenging  concepts  increases  the 
remaining  available  time  for  collaboratively  processing  those  concepts  in  online  discussions  (see  Kirschner  et  al.,  2009  for  a  similar  argu¬ 
ment).  Another  caveat,  as  noted  by  Lowe  (2004),  is  that  cues  may  prompt  students  to  process  information  in  isolated  ways  without 
addressing  the  overall  aspects  of  a  learning  task.  Thus,  a  central  challenge  in  attention  guidance  is  to  introduce  the  support  without 
destroying  the  exploratory  and  creative  potential  of  collaborative  knowledge  construction.  To  move  toward  this  direction,  the  following 
section  will  explain  the  design  of  two  different  attention  guidance  functionalities  in  an  anchored  discussion  system. 

3.  Design  of  attention  guidance  functionalities 

We  designed  three  prototype  software  systems  by  further  developing  Van  der  Pol  et  al.’s  open  source  annotation  tool  (2006).  We 
employed  this  tool  for  two  reasons.  First,  it  has  a  user-friendly  interface  that  provides  a  tight  coupling  between  the  instructional  material 
and  its  related  discussion  without  hindering  the  interaction  among  students  (e.g.,  Eryilmaz,  Alrushiedat,  Kasemvilas,  Mary,  &  Van  der  Pol, 
2009;  Eryilmaz,  Van  der  Pol,  Clark,  Mary,  &  Ryan,  2010;  Eryilmaz,  Van  der  Pol,  Kasemvilas,  Mary,  &  Olfman,  2010).  Second,  this  tool  reduces 
explicit  coordination  activities  during  online  social  interaction  and  thereby  availing  students  more  time  and  effort  for  demanding 
knowledge  construction  activities  that  positively  associate  with  individual  learning  outcomes  (Eryilmaz,  Van  der  Pol,  Ryan,  Clark,  &  Mary, 
2013). 

All  three  of  the  prototypes  capitalize  on  Marginalia,  an  open  source  JavaScript  program,  for  the  fine-grained  annotation  of  HTML  pages. 
Marginalia  allows  users  to  create  new  annotations  by  selecting  desired  text  with  the  mouse  and  then  clicking  on  an  annotation  bar  to  the 
right  of  the  instructional  material.  Additionally,  Marginalia  supports  multiple  annotations  on  desired  text  to  maintain  independent  threaded 
discussions  that  focus  on  the  same  topic.  The  flexibility  of  the  HTML  interface  allows  inclusion  of  two  new  functional  characteristics  in  all 
prototypes.  These  functional  characteristics  aim  to  provide  a  stronger  link  between  the  instructional  material  and  its  related  discussion  as 
this  linkage  leads  to  superior  learning  outcomes  compared  to  systems  that  lack  it  (Eryilmaz,  Van  der  Pol,  et  al„  2013).  More  specifically,  the 
first  functional  characteristic  lights  up  both  the  annotated  text  and  corresponding  threaded  discussion  in  red  when  either  element  is  under 
cursor.  The  second  functional  characteristic  embeds  a  student’s  key  idea  or  justification  for  making  an  annotation  directly  in  the  relevant 
context  that  elicited  it  by  inserting  a  sticky  message.  Therefore,  the  second  functional  characteristic  acts  as  a  catalyst  to  distill  students’  key 
ideas  when  annotating  instructional  resources.  Furthermore,  each  threaded  discussion  starts  with  a  student  submitted  key  idea,  which 
focuses  on  a  specific  content  to  discourage  topic  drift.  Subsequently,  we  marked  the  particular  point  of  each  threaded  discussion  with  a 
student  submitted  key  idea  and  marked  peers’  uptakes  of  an  original  key  idea  as  statements  in  threaded  discussions.  The  flipside  of  an 
interface  design  with  sticky  messages  is  that  it  may  interfere  with  students’  reading  as  the  instructional  material  becomes  cluttered  with 
sticky  messages.  To  solve  this  problem,  we  designed  sticky  messages  to  appear  only  under  the  cursor.  In  sum,  all  three  prototypes  promote 
context-based  discussions  for  deep  processing  of  complex  instructional  materials.  However,  they  differ  on  the  nature  of  the  attention 
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guidance  functionalities  as  described  below.  Importantly,  attention  guidance  functionalities  described  below  go  beyond  providing  students 
annotated  instructional  materials  in  anchored  discussions  (Wolfe,  2008)  because  students  still  have  to  distill  the  key  ideas  on  their  own 
when  annotating  difficult  passages  from  text. 

3.1.  Instructor-based  attention  guidance  functionality 

The  first  prototype  (Fig.  1)  includes  an  instructor  interface  to  find  a  balance  between  offering  students  an  inherently  open  learning 
environment  where  they  can  express  their  own  ideas  and  steering  them  toward  discussing  challenging  concepts  from  complex  instructional 
materials  (Kirschner,  Sweller,  &  Clark,  2006).  In  this  prototype,  only  the  instructor  dynamically  alters  text  font  size  by  selecting  desired  text 
with  the  mouse  and  then  clicking  on  an  importance  bar  to  the  left  of  the  instructional  material.  Thus,  the  importance  bar  is  not  visible  to 
students  in  this  prototype.  Font  size  as  demonstrated  in  tag  cloud  research  is  an  effective  visual  property  to  capture  attention  (e.g.,  Bateman, 
Gutwin,  &  Nacenta,  2008;  Halvey  &  Keane,  2007;  Lohmann,  Ziegler,  &  Tetzlaff,  2009;  Rivadeneira,  Gruen,  Muller,  &  Millen,  2007).  Moving 
this  line  of  research  forward,  the  first  prototype  represents  the  importance  of  the  presented  information  with  a  corresponding  font  size.  The 
cascading  style  sheet  of  the  first  prototype  includes  two  font  sizes:  default  and  big.  On  the  one  hand,  the  default  font  size  (lOpx)  represents  a 
medium  level  of  importance.  On  the  other  hand,  the  big  font  size  represents  a  high  level  of  importance  determined  by  the  instructor-based 
on  the  learning  objectives  of  social  interaction.  In  order  to  determine  which  big  font  size  captures  attention  in  an  involuntary  or  obligatory 
fashion,  we  created  several  font  sizes  ranging  from  11  px  to  25px.  We  did  not  go  above  25px  because  of  the  limited  real  estate  available  in  the 
margins  of  the  learning  material.  To  aid  in  the  design  specification  stage,  we  asked  a  separate  group  of  pilot  study  participants  to  identify  and 
report  the  font  size  that  exerted  the  most  visual  influence  on  their  attention.  Based  on  their  input,  we  set  the  big  font  size  to  be  150%  larger 
than  the  default  font  size. 


3.2.  Peer-oriented  attention  guidance  functionality 


4.  Research  questions  and  hypotheses 

Despite  the  potential  of  attention-guiding  cues  for  deep  processing  of  challenging  concepts  from  complex  instructional  materials,  only 
few  empirical  studies  exist  in  the  CSCL  literature  about  the  effects  of  such  support  in  online  discussions  (for  a  review  see  Wecker  &  Fischer, 
2007).  Furthermore,  most  studies  do  not  provide  direct  evidence  on  students’  perception  of  a  complex  instructional  material’s  challenging 
concepts  and  the  quality  of  collaboratively  constructed  knowledge  focusing  on  those  concepts  (see  Sereno,  Shum,  &  Motta,  2005  for  the 
design  and  evaluation  of  a  rhetorical  parser  that  assesses  a  document’s  important  sentences  by  checking  whether  each  of  its  words  appears 
in  the  title,  the  headers,  or  the  abstract).  Drawing  on  the  hefty  cost  of  discovering  a  new  understanding  together  that  no  one  in  discussion  yet 
possesses  and  the  positive  learning  outcomes  facilitated  by  an  earlier  version  of  the  relevant  anchored  discussion  system  (Eryilmaz,  Van  der 
Pol,  et  al.,  2013),  this  study  tacldes  the  following  research  questions  and  their  accompanying  hypotheses: 
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Fig.  3.  Control  software  system. 

I.  How  does  an  anchored  discussion  system’s  instructor-based  attention  guidance  functionality  affect  collaborative  knowledge  con¬ 
struction  focusing  on  challenging  concepts  from  complex  instructional  materials? 


Collaboratively  constructing  new  knowledge  that  impacts  learning  more  directly  entails  selecting  challenging  concepts  from  complex 
instructional  materials  that  subsequently  serve  as  the  focus  of  intellectual  advances  in  areas  where  students  struggle  to  understand.  For 
example,  in  the  subject  of  constructivist  instruction,  such  concepts  and  principles  may  include  breaking  down  instruction  with  the  intention 
of  allowing  students  to  more  easily  understand  the  content  and  guiding  students  through  a  series  of  small  steps,  carefully  fit  together,  to  the 
appropriate  discoveries  (“Aha!”  experience).  CSCL  research  on  constructivist  learning  has  repeatedly  shown  that  students  with  low  prior 
domain  knowledge  need  guidance  allowing  them  to  learn  challenging  concepts  more  effectively  and  efficiently  in  complex  domains 
(Hmelo-Silver,  Duncan,  &  Chinn,  2007;  Kirschner  et  al.,  2006).  We  consider  the  instructor-based  attention  guidance  functionality  as 
instructional  scaffolding  for  students  with  low  prior  knowledge  in  order  to  steer  them  toward  discussing  challenging  concepts  in  an  open 
learning  environment.  Furthermore,  students  may  also  challenge  an  instructor’s  views  in  this  form  of  guidance,  which  we  find  positive.  For 
instance,  a  student  may  post  “I  do  not  see  why  this  matters”,  which  leads  to  further  negotiations.  Thus,  we  hypothesize: 

HI  a:  Instructor-based  attention  guidance  functionality  in  anchored  discussion  system  will  facilitate  students’  selection  of  challenging 
concepts  from  complex  instructional  materials. 


If  a  collaborative  knowledge  construction  task  must  be  finished  within  a  certain  time-limit,  then  the  time  dedicated  to  sharing  existing 
opinions  and  experiences  reduces  the  time  available  for  diagnosing  and  resolving  challenging  misconceptions  from  complex  instructional 
materials.  Furthermore,  when  students  gravitate  to  familiar  (comfortable)  topics,  online  discussions  tend  to  branch  and  progressively 
become  more  incoherent  (Hewitt,  2005;  Lipponen  et  al.,  2003;  Suthers,  Vatrapu,  Medina,  Joseph,  &  Dwyer,  2008).  The  coherence  problem 
undermines  the  educational  value  of  collaborative  knowledge  construction  as  students  engage  into  many  parallel  conversations,  not  all  of 
which  necessarily  focus  on  areas  where  they  struggle  to  understand.  Students’  difficulties  with  preserving  the  interactional  coherence  of 
online  discussions  are  understandable  because  everything  may  look  important  to  them  when  they  try  to  make  sense  of  a  complex 
instructional  material  (see  Foss,  1989  for  a  similar  argument).  Thus,  systems  that  help  students  focus  on  challenging  concepts  can  improve 
their  social  interactions  and  their  learning.  This  leads  to  the  following  hypothesis: 


Hlb:  Instructor-based  attention  guidance  functionality  in  an  anchored  discussion  system  will  increase  the  quantity  or  improve  the 
patterns  of  collaborative  knowledge  construction  activities  focusing  on  challenging  concepts  from  complex  instructional  materials. 

II.  After  switching  from  an  instructor-based  to  peer-oriented  attention  guidance  functionality  in  an  anchored  discussion  system,  will 
students  continue  to  engage  in  high  quality  social  interaction  patterns  focusing  on  challenging  concepts  from  complex  instructional 
materials? 
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The  underlying  assumption  of  the  foregoing  argumentation  is  that  an  instructor’s  guidance  can  help  students  with  low  prior  domain 
knowledge  to  learn  challenging  concepts  from  complex  instructional  materials.  However,  as  students  become  more  familiar  with  a  subject 
matter,  instructor’s  guidance  may  become  redundant  and  in  some  cases  it  may  even  hinder  collaborative  knowledge  construction  because 
students  may  rely  too  much  on  an  instructor’s  expertise  and  authority  (Puntambekar  &  Hubscher,  2005).  Therefore,  the  notion  of  successful 
scaffolding  entails  the  fading  of  an  instructor’s  guidance,  which  forces  students  to  practice  their  own  knowledge  and  develop  the  necessary 
skills  for  novel  situations  (Wecker  &  Fischer,  2007).  Although  fading  is  considered  important  within  the  scaffolding  paradigm,  empirical 
research  results  on  the  effects  of  fading  are  sparse  and  inconclusive  (for  a  review  see  Bouyias  &  Demetriadis,  2012).  The  issue  of  fading 
becomes  more  challenging  in  computerized  scaffolding  because  it  requires  designing  dynamic  software  systems  that  provide  adaptive 
support  (Kim  &  Hannafin,  2011).  We  argue  that  if  students  with  low  prior  domain  knowledge  internalize  the  learning  task  they  are  being 
helped  to  accomplish  via  an  instructor-based  attention  guidance  functionality,  they  will  take  the  lead  in  pinpointing  challenging  concepts 
they  did  not  previously  understand  via  a  peer-oriented  attention  guidance  functionality.  Thus,  we  hypothesize: 

H2:  After  switching  from  an  instructor-based  to  peer-oriented  attention  guidance  functionality  in  an  anchored  discussion  system, 
students  will  continue  engaging  in  high  quality  social  interaction  patterns  focusing  on  challenging  concepts  from  complex  instructional 
materials. 


5.  Method 

The  evaluation  of  the  proposed  attention  guidance  functionalities  included  two  stages:  a)  pilot  study  conducted  to  determine  which  big 
font  size  captured  attention  in  an  involuntary  or  obligatory  fashion,  b)  main  study  aimed  to  test  the  above  hypotheses.  Pilot  study  par¬ 
ticipants  were  8  freshmen  doctoral  students  (7  males  and  1  female)  from  a  blended-format  seminar  in  information  systems  and  technology 
research.  The  mean  age  of  the  pilot  study  participants  was  25.74  years  (SD  =  2.36).  The  learning  objective  of  this  seminar  was  to  introduce 
students  to  the  community  of  information  systems  and  technology  research,  its  participants,  the  topics  they  investigate,  and  the  approaches 
they  use.  Six  discussion  themes  were  run  consecutively  through  the  seminar  for  a  period  of  one  week  each.  These  discussion  themes  were 
facilitated  by  the  instructor-based  attention  guidance  functionality.  Each  theme  reflected  a  specific  research  article  from  the  Management 
Information  Systems  Quarterly  journal.  At  the  end  of  each  discussion  theme,  the  main  researcher  conducted  face-to-face  interviews  by 
asking  pilot  study  participants  the  order  through  which  they  processed  the  contents  of  a  learning  material.  The  interviewees  remarked  that 
sentences  with  150%  larger  font  size  “pop  out”  from  their  surroundings,  which  led  interviewees  to  process  them  first,  even  if  they  were  no 
more  likely  to  be  important  than  any  other  sentence. 

The  main  study  was  a  longitudinal  experiment  with  two  small  groups:  treatment  and  control.  We  specifically  chose  this  granularity 
because  it  allowed  a  wide  range  of  social  interactions  to  play  out  while  preventing  us  from  losing  track  of  them  (Stahl,  2006).  Students  in  the 
control  group  had  no  access  to  attention  guidance  functions.  Students  in  the  treatment  group  used  instructor-based  attention  guidance 
functionality  and  then  switched  to  peer-oriented  attention  guidance  functionality.  The  longitudinal  nature  of  the  study  provided  the  means 
for  calibrating  the  guidance  to  students  in  the  treatment  group  through  ongoing  diagnosis  of  online  discussions.  This  ongoing  diagnosis 
included  two  variables:  topic  of  a  message  and  activity  performed  within  a  message.  Two  independent  coders  who  were  blind  to  the  study’s 
purpose  examined  these  variables  after  students  completed  each  discussion  topic.  This  approach,  which  underscores  an  advantage  of  small 
group  experimentation  (Stahl,  2006),  allowed  us  to  switch  the  treatment  group  from  instructor-based  to  peer-oriented  attention  guidance 
functionality  in  order  to  test  H2.  Participants  were  24  doctoral  students  (8  males  and  16  females)  from  a  blended-format  seminar  in  Learning 
and  Pedagogical  Theories.  The  mean  age  of  the  students  was  26.38  years  (SD  =  1.98).  We  randomly  and  equally  assigned  the  participants  to 
one  of  two  groups.  The  learning  material  for  both  groups  consisted  of  six  topics  from  the  Constructivist  Instruction:  Success  or  Failure  book. 
These  topics  were  arranged  in  the  following  sequence.  Topic  1  was  “The  success  or  failure  of  constructivist  instruction”;  topic  2  was  “Taking 
guided  learning  theory  to  school”;  topic  3  was  “Epistemology  or  pedagogy,  that  is  the  question”;  topic  4  was  “The  empirical  support  for 
direct  instruction";  topic  5  was  “from  Behaviorism  to  constructivism:  a  philosophical  journey  from  drill  and  practice  to  situated  learning"; 
and  topic  6  was  “Beyond  the  fringe:  building  and  evaluating  scientific  knowledge  systems.”  Each  topic  was  covered  during  a  two-week 
online  discussion  period  in  an  inherently  open  learning  environment.  Participation  in  the  online  discussion  was  compulsory  and  repre¬ 
sented  25%  of  the  overall  grade.  The  minimum  participation  requirement  was  to  post  two  messages  per  topic  and  respond  to  at  least  two 
fellow  students’  messages  for  that  topic.  When  using  the  peer-oriented  attention  guidance  functionality,  every  student  in  the  treatment 
group  was  additionally  asked  to  use  the  importance  bar  at  least  once  in  order  to  collaboratively  decide  important  ideas  from  text.  The  main 
body  of  gathered  data  included  the  transcripts  of  12  online  discussions.  We  describe  below  the  methods  of  analyses  used  to  test  the 
hypotheses. 

5.1.  Analyzing  students’  selection  of  challenging  concepts 

Annotations  as  described  by  Yeh  and  Lo  (2009)  provide  insight  into  the  focus  of  students’  attention  on  text.  We  calculated  a  selection 
ratio  for  each  group’s  discussion  topic  by  dividing  student  annotations  on  challenging  concepts  (e.g.,  designing  learning  activities  that 
challenge  students’  current  suppositions  and  distinguishing  scientific  claims  from  pseudoscience  and  hoax  claims)  to  total  student  anno¬ 
tations.  We  used  the  first  five  discussion  themes  for  the  analysis  of  instructor-based  attention  guidance  functionality.  We  computed 
summary  statistics  (mean,  standard  deviation)  and  tested  for  significant  differences  between  the  instructor-based  attention  guidance 
functionality  and  control  condition  with  two-tailed  t-tests  and  proportion  tests  (Kennedy,  2008).  In  light  of  our  findings,  we  examined  the 
effects  of  switching  treatment  group  students  from  instructor-based  to  peer-oriented  attention  guidance  functionality  on  the  last  discussion 
topic. 

In  order  to  verify  the  effectiveness  of  the  proposed  attention  guidance  functionalities,  we  constructed  visual  heat  maps  based  on  stu¬ 
dents’  cursor  movements  during  the  course  of  six  discussion  topics  (see  Chen,  Anderson,  &  Sohn,  2001  for  a  strong  positive  correlation 
between  cursor  movement  and  attention).  The  standardized  heat  maps  employed  ClickTale  Web  application’s  tracking  code.  We 
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incorporated  the  relevant  code  into  an  HTML  interface,  and  ran  each  discussion  topic’s  tracking  for  two  weeks.  The  colors  on  the  heat  maps 
ranged  from  red  to  blue.  Red  color  suggests  the  areas  that  received  the  most  student  attention;  the  yellow  color  suggests  areas  that  received 
less  student  attention;  and  blue  color  suggests  areas  that  received  the  least  student  attention  during  online  discussions. 


5.2.  Analyzing  students’  discussion  of  challenging  concepts 

This  study  employed  quantitative  content  analysis,  sequential  analysis,  and  statistical  discourse  analysis  to  gain  insight  into  discussions 
focusing  on  challenging  concepts  that  students  struggle  to  understand.  We  adopted  a  coding  scheme  developed  by  Li  and  Huang  (2008)  to 
code  online  discussion  messages.  The  unit  of  content  analysis  was  each  complete  message  posted  in  the  online  discussion  because  students’ 
messages  were  rather  short  and  mainly  consisted  of  only  one  type  of  activity  (see  Eryilmaz,  Ryan,  et  al.,  2013  for  a  similar  argument).  Table  1 
describes  the  coding  categories  with  examples  of  raw  data  for  each  category. 

We  examined  two-event  sequences  of  threaded  discussions  with  sequential  analysis.  A  central  underlying  premise  of  this  analytic 
framework  is  that  messages  in  a  threaded  discussion  are  inherently  interconnected  and  dynamically  affect  one  another  (Lu,  Chiu,  &  Law, 
2011;  Suthers,  Dwyer,  Medina,  &  Vatrapu,  2010;  Wise  &  Chiu,  2011).  Thus,  a  message  within  a  threaded  discussion  can  only  be  fully  un¬ 
derstood  by  examining  its  micro-context  of  preceding  messages  (Chiu,  2008).  We  used  a  discussion  analysis  tool  (DAT)  developed  byjeong 
and  Frazier  (2008)  to  analyze  two-event  sequences  in  threaded  discussions.  DAT  offers  two  metrics.  The  first  metric,  a  transitional  state 
diagram,  provides  an  overview  of  the  flow  of  interaction  patterns.  DAT  calculates  transitional  probabilities  by  tallying  the  frequency  of 
responses  for  each  message  type.  The  second  metric,  mean  response  scores,  computes  the  relative  frequency  of  each  type  of  message 
eliciting  a  particular  type  of  response.  Thus,  the  second  metric  is  well  suited  for  comparing  the  differences  between  probabilities  of 
transitions. 

However,  a  limitation  with  two-event  sequence  analysis  of  threaded  messages  as  noted  by  Suthers  et  al.  (2010)  is  that  it  models  a  current 
message  based  on  one  previous  message,  even  though  a  message  may  depend  on  a  cumulative  series  of  immediately  preceding  messages 
(see  Zemel,  Xhafa,  &  Cakir,  2007  for  a  similar  argument).  To  address  this  limitation,  we  conducted  a  statistical  discourse  analysis  (SDA)  (Chiu, 
2008)  to  examine  longer  sequences  of  threaded  messages.  Statistically  analyzing  social  interaction  processes  involves  difficulties  regarding 
the  dataset,  outcome  variables,  and  exploratory  variables.  SDA  (Chiu,  2008)  addressed  these  issues.  We  use  Markov  Chain  Monte  Carlo 
multiple  imputation  (MCMC-M1)  to  estimate  the  values  of  the  missing  data,  which  addresses  this  issue  more  effectively  than  deletion,  mean 
substitution,  or  simple  imputation  according  to  computer  simulations  (Peugh  &  Enders,  2004).  SDA  addresses  the  outcome  issues  (nested 
data,  time,  discrete,  infrequent,  multiple  outcomes)  with  multilevel  analysis  (Goldstein,  1995),  an  I2  index  of  Q-statistics  (Huedo-Medina, 
Sanchez-Meca,  Marin-Martinez,  &  Botella,  2006),  Logit,  Logit  bias  estimation  (King  &  Zeng,  2001)  and  a  multivariate  outcome  model 
(Goldstein,  1995).  SDA  addresses  the  explanatory  variable  issues  (sequences,  context-dependent  effects,  indirect  effects,  false  positives)  with 
a  vector  auto-regression  (VAR,  Kennedy,  2008),  multilevel  random  effects,  multilevel  M-tests  (MacKinnon  et  al„  2004)  and  the  two-stage 
step-up  procedure  (Benjamini,  Krieger,  &  Yekutieli,  2006). 


5.3.  Analyzing  control  variables 

We  included  prior  knowledge  and  task  complexity  as  the  most  important  control  variables  in  our  study  (a  thorough  discussion  of  a  wide 
variety  of  factors  is  beyond  the  scope  of  this  article  see  Kirschner,  Martens,  &  Strijbos,  2004  for  complex  set  of  cognitive,  motivational,  and 
social  factors).  Empirical  research  focusing  on  conceptual  change  induced  by  text  reading  has  shown  that  students’  prior  knowledge  affects 
how  they  interact  with  the  instructional  materials  (Ozuru,  Dempsey,  &  McNamara,  2009).  Particularly  pertinent  here  is  that  high  prior 
knowledge  can  facilitate  the  cognitive  processing  of  reading  (Wolfe  &  Goldman  2005).  In  contrast,  low  topic  knowledge  often  confounds 
students’  best  efforts  to  understand  text  accurately  (Sinatra  &  Mason,  2008).  Before  participating  in  the  learning  activities,  each  student’s 
prior  knowledge  of  the  subject  domain  was  assessed  with  an  open-ended  question:  “Briefly  describe  inquiry-based  constructivist  in¬ 
struction  in  information  systems  and  technology  education.  What  are  the  strengths  and  weaknesses  of  engaging  students  in  open-ended 
investigations?  Provide  reasoning  for  each  strength  and  weakness.”  To  answer  this  question,  each  student  had  20  minutes  to  write  a 
short  reflective  essay  alone  without  consulting  any  resources.  Using  an  essay  coding  framework  developed  by  Jamaludin,  Chee,  and  Ho 
(2009),  two  trained  coders  independently  scored  each  essay  from  0  to  12  points  without  knowing  each  student’s  assigned  condition. 


Elaborating 

Negotiating 

Producing 


Description 

Presenting  an  initial  interpretation 
of  learning  material 

Discovering  dissonance  or  inconsistency 
among  ideas 

Expanding  ideas  from  earlier  messages 
in  a  discussion  thread 

Comparing  and  contrasting  different 
perspectives  for  a  deeper  understanding 
of  learning  material 

Producing  new  ideas  or  explanations  on 
topics  where  differences  in  perspectives 
existed 


Example 

I  think  the  author  aims  to  strengthen  the  argument  for  a  systematic  pattern  of  direct  instruction 
as  a  primary  tool  for  effective  teachers  by  differentiating  between  epistemology  and  pedagogy. 

The  question  that  kept  arising  for  me  from  your  message  was  whether  or  not  this  approach  is  a  one-size- 
fits-all  across  various  disciplines.  This  seems  obvious  to  me  when  it  comes  to  subjects  like  math,  science, 
and  reading,  but  I  am  intrigued  by  constructivist  assertions  that  not  all  knowledge  is  task-based. 

1  appreciate  you  bringing  up  the  movement  from  novice  to  expert  My  hunch  is  that  somewhere  along 
that  journey,  hands-on,  practical  experimentation  becomes  an  important  part  of  an  expert's 
development  of  foundational  knowledge. 

I  would  argue  that  mistakes  are  necessary  in  the  process  of  discovery  based  learning  because  they 
become  a  further  teaching  opportunity:  unraveling  presuppositions  by  asking  the  right  question, 
introducing  new  evidence,  etc. 

Cot  it  On  an  elementary  level  there  are  curriculum  books  for  teaching  creativity.  See  critical  thinking 
skills  for  grades  1  -2  published  by  Evan-Moor.  1  just  checked  online  and  they  renamed  their  series  critical 
and  creative  thinking  skills.  I  think  creativity  can  be  taught. 
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The  challenges  of  a  learning  task  naturally  influence  the  dynamics  of  online  discussion.  Previous  research  stressing  the  potential  of  this 
social  interaction  process  shows  that  collaborative  learning  is  more  effective  than  individual  learning  for  relatively  complex  tasks  (Kirschner 
et  al.,  2009).  However,  a  task  that  is  too  difficult  can  also  undermine  the  social  interaction  process  because  students  can  get  frustrated  and 
lose  their  motivation  (Timmers  &  Veldkamp  2011).  Thus,  similar  to  Vygotsky’s  (1978)  concept  of  the  zone  of  proximal  development,  an 
optimal  task  complexity  lies  just  above  what  individual  learners  can  accomplish  alone.  We  operationalized  task  complexity  by  using  a 
labeled  six-point  Likert-type  subjective  rating  scale  validated  by  Cierniak,  Scheiter,  and  Gerjets  (2009).  The  labels  reached  from  1  “Not  at  all” 
to  6  “Extremely.”  The  question  of  the  subjective  rating  scale  was  “How  difficult  was  the  learning  task  for  you?”  The  rationale  behind  using  a 
subjective  rating  scale  is  twofold.  First,  prior  research  has  demonstrated  that  unobtrusive  subjective  rating  scales  are  more  sensitive  to 
variations  in  instructional  tasks  than  physiological  measures  (Paas,  Van  Merrienboer,  &  Adam,  1994).  Second,  subjective  ratings  as  noted  by 
Schnotz  and  Kurschner  (2007)  are  simple  and  easily  applicable  in  a  natural  setting,  which  increases  the  ecological  validity  of  the  findings. 

6.  Results 

The  presentation  of  the  results  follows  the  order  of  our  hypotheses.  First,  we  present  the  effect  of  instructor-based  attention  guidance 
functionality  on  selecting  challenging  concepts  from  complex  instructional  materials.  Thereafter,  we  show  the  coding  of  students’  posts  and 
patterns  in  online  discussions  focusing  on  challenging  concepts.  Finally,  we  present  the  focus,  coding,  and  sequential  organization  of  the 
treatment  group’s  online  discussion  after  they  switch  from  the  instructor-based  to  peer-oriented  attention  guidance  functionality. 

6.7.  Students'  selection  of  challenging  concepts 

HI  a  predicted  that  instructor-based  attention  guidance  functionality  in  an  anchored  discussion  system  would  facilitate  students’  se¬ 
lection  of  challenging  concepts  from  complex  instructional  materials.  We  computed  a  selection  ratio  of  challenging  concepts  for  each 
discussion  topic.  Table  2  shows  that  treatment  group  students  had  both  a  greater  frequency  of  annotations  focusing  on  challenging  concepts, 
t(8)  =  3.11,  p  =  0.02,  d  — 1.97,  and  a  greater  proportion  of  annotations  focusing  on  challenging  concepts,  z  =  5.1,  p  <  0.001,  than  the  control 
group. 

While  the  above  statistical  analysis  tested  whether  students’  selection  of  challenging  concepts  was  differed  significantly  between  the 
two  groups,  it  did  not  provide  information  about  how  students  viewed  cued  and  non-cued  passages  from  text.  This  important  information 
complementary  to  statistical  analysis  is  available  through  heat  maps.  The  heat  maps  exposed  that  instructor’s  cues  had  an  impact  on  the  way 
students’  attention  was  distributed  in  text.  Fig.  4  presents  a  heat  map  derived  from  ten  page  views  by  ten  different  students  in  the  treatment 
group.  This  heat  map  suggests  that  students  paid  the  most  attention  to  understanding  difficult  passages  suggested  by  the  instructor  via  big 
font  size  and  sticky  messages  summarizing  their  own  key  ideas  for  annotating  those  passages,  as  indicated  by  the  red  color  (in  the  web 
version). 

Fig.  5  shows  a  heat  map  derived  from  eight  page  views  by  eight  different  students  in  the  control  group.  A  striking  feature  of  this  heat  map 
is  that  students'  attention  was  more  widely  distributed  on  a  page,  but  less  concentrated  in  any  particular  area,  as  indicated  by  the  yellow  and 
green  colors  (in  the  web  version).  This  heat  map  also  suggests  that  sticky  messages  that  distill  students’  key  ideas  for  making  annotations 
received  less  attention  when  the  entire  text  had  the  default  font  size.  Taken  together,  these  heat  maps  provide  further  support  to  HI  a. 

6.2.  Discussing  task-relevant  information  findings 

Hlb  predicted  that  instructor-based  attention  guidance  functionality  in  an  anchored  discussion  system  would  increase  the  quantity  or 
improve  the  patterns  of  collaborative  knowledge  construction  activities  focusing  on  challenging  concepts  from  complex  instructional 
materials.  As  noted  before,  the  results  of  discussing  challenging  concepts  from  complex  instructional  materials  incorporate  two  parts: 
classifying  individual  messages  and  examining  sequential  relationships  in  threaded  discussions.  The  dataset  for  testing  Hlb  included  the 
transcripts  of  the  first  five  discussion  topics.  Because  of  the  potential  for  ambiguity  in  applying  the  coding  scheme,  a  random  sample  of  36 
messages  (10%  of  the  total)  was  selected  for  the  training  of  two  coders  who  were  blind  to  our  hypotheses.  After  the  training  exercise,  a  total 
of  448  messages  were  coded  independently  by  each  individual  coder.  The  Krippendorff  s  alpha  (2004)  inter-rater  reliability  coefficient  was 
0.78  which  exceeds  0.67  and  indicates  satisfactory  agreement  beyond  chance.  All  disagreements  between  coders  were  resolved  by  dis¬ 
cussion.  Table  3  displays  coding  results  at  a  discussion  level  according  to  the  variables  of  interest. 


ratio  of  challenging  concepts  I 


Condition 


first  five  discussion  topics. 


Discussion  topic 
1  2  3 


Instructor-based  attention  guidance  functionality 
Frequency  of  student  annotations  focusing  on  challenging  concepts  9  12 

Total  number  student  annotations  14  17 

Selection  ratio  of  challenging  concepts  64%  70% 

Control  software  system 

Frequency  of  student  annotations  focusing  on  challenging  concepts  3  8 

Total  number  of  student  annotations  9  19 

Selection  ratio  of  challenging  concepts  33%  42% 


Note.  Standard  deviation  in  parenthesis. 


31% 
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Fig.  4.  Heat  map  for  teacher-based  attention  guidance  functionality. 


We  found  that  the  treatment  group  had  a  higher  proportion  of  task-related  messages  than  the  control  group,  z  =  5.43,  p  <  0.001.  Closer 
examination  demonstrated  that  treatment  group  produced  proportionally  more  questions,  elaborations,  and  negotiations  than  the  control 
group,  with  no  significant  differences  in  sharing  or  producing  (see  Table  3). 

The  above  quantitative  content  analysis  results  served  as  a  foundation  for  examining  prevalent  patterns  of  collaborative  knowledge 
construction  activities  through  two-  and  three-event  sequences  of  students’  messages  in  threaded  discussions.  Thus,  the  validity  of 
sequential  analyses  results  reported  below  tied  directly  to  the  validity  of  previous  message  categorization.  For  the  first  five  discussion  topics, 
the  average  discussion  thread  length  for  the  treatment  group  was  2.93  (SD  =  2.32),  while  the  average  discussion  thread  length  for  the 
control  group  was  4.16  (SD  =  3.02),  t(88)  =  2.16,  p  =  0.03,  d  =  0.46.  For  the  two-event  sequence  analysis,  all  303  task-related  messages  (174 
for  the  treatment  group  and  129  for  the  control  group)  underwent  a  series  of  sequential  transition  matrix  calculations  (Jeong  &  Frazier, 
2008),  resulting  in  Figs.  6  and  7,  and  Table  4.  The  circles  in  these  figures  represent  the  task-related  messages,  in  accordance  with  the 
adopted  coding  scheme.  The  arrows  between  the  circles  depict  directed  transitions.  The  thickness  of  an  arrow  is  proportional  to  probability 
of  a  transition.  The  values  specified  in  the  figures  illustrate  transitional  probabilities.  These  probabilities  are  based  on  the  frequency  of  a 
particular  response  posted  in  reply  to  a  particular  message  category.  For  instance,  a  sharing  message  category  was  followed  by  an  elabo¬ 
rating  response  49%  of  the  time  with  the  instructor-based  attention  guidance  functionality.  To  ease  readability,  transitional  probabilities  less 
than  0.1  were  omitted  in  Figs.  6  and  7. 

Table  4  reports  descriptive  statistics  for  the  examination  of  two-event  sequences  in  threaded  discussions.  There  are  three  metrics  in  Table 
4:  sample  size,  mean,  and  standard  deviation.  The  first  metric,  sample  size,  indicates  the  total  number  of  times  a  particular  response 
immediately  follows  a  particular  message  category.  As  shown  in  Table  4,  elaboration  messages  immediately  followed  sharing  messages  64 
times  with  the  instructor-based  attention  guidance  functionality.  The  second  metric,  mean,  represents  the  mean  number  of  a  particular 
response  posted  in  reply  to  a  particular  message  category.  For  example,  the  mean  number  of  elaboration  messages  posted  in  reply  to  each 
sharing  message  was  0.38  with  the  instructor-based  attention  guidance  functionality.  The  last  metric,  standard  deviation,  expresses  the 
variability  among  instances  of  a  particular  message  category  to  elicit  a  specific  response. 
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Fig.  5.  Heat  map  for  control  software  system. 


We  found  that  the  students  in  the  treatment  group  demonstrated  significantly  higher  transitional  probabilities  from  sharing  to  elab¬ 
orating,  t(137)  =  2.56,  p  =  0.01,  d  =  0.44,  and  from  questioning  to  negotiating,  t(84)  =  2.76,  p  =  0.007,  d  =  0.63,  compared  to  control  group 
students.  To  facilitate  the  explanation  of  these  sequential  patterns,  we  draw  upon  two  discussion  samples  from  the  treatment  group  below. 
The  first  discussion  sample  (see  Table  5)  stems  from  “epistemology  or  pedagogy,  that  is  the  question"  topic  studied  in  the  course.  The 
starting  point  of  this  discussion  presents  student  7’s  interpretation  of  the  relationship  between  epistemology  and  pedagogy  expressed 
through  an  analogy:  “standards  and  knowledge  drive  instruction.”  Student  12’s  response  to  this  message  reflects  a  shared  understanding  of 
the  core  concepts:  epistemology  and  pedagogy.  Subsequently,  student  12  moved  student  7’s  initial  idea  forward  by  articulating  that 
“pedagogy  is  not  one-size-fits-all.” 

Table  3 

Content  analysis  of  first  five  discussion  topics. 


Messages  Instructor-based  attention  Control  software  system  z 

guidance  functionality 

Frequency  Proportion  Frequency  Proportion 


Task-related  Sharing  66 

Elaborating  30 

Negotiating  23 

Producing  3 

Total  task-related  174 


0.30 

0.24 

0.14 


0.01 

0.80 


76  0.33  0.68 

36  0.16  2.12* 

11  0.04  3.73*** 

5  0.02  3.90*** 

1  0.004  0.77 

129  0.56  5.43*** 


Non  task-related 


43 


0.20  102  0.44 


0.50 

0.03 

<0.001 

<0.001 


*p  <  0.05,  **p< 


:  0.01,  ***p<  0.001. 


Fig.  6.  Transitional  state  diagram  for  treatment  group  *p  <  0.05;  **p  <  0.01.  Note:  transitional  probabilities  less  than  0.1  are  omitted. 

The  second  discussion  sample  (Table  6)  stems  from  “the  empirical  support  for  direct  instruction"  topic  studied  in  the  course.  The 
beginning  of  this  discussion  underscores  student  l’s  confusion  from  text,  which  raises  two  though-provoking  questions  on  the  concept  of 
“process-product.”  An  attempt  to  answer  these  questions  comes  from  student  8  whose  argumentation  centers  on  learning  goals  in  order  to 
examine  explicit  instruction  and  constructivist  methods  in  information  systems  education.  Consistent  with  student  1,  student  8  highlights  a 
knowledge  gap  at  the  end  of  message  number  90  to  invite  further  negotiation  on  this  complicated  topic. 

As  mentioned  before,  we  assessed  group  differences  in  three-event  sequences  within  threaded  discussions  by  using  SDA  (Chiu,  2008). 
Consistent  with  the  above  analyses,  we  found  that  the  treatment  group  was  more  likely  than  the  control  group  to  elaborate.  Likewise, 
consistent  with  the  sequential  analysis,  sharing  was  often  followed  by  elaboration  and  questioning  was  often  followed  by  negotiation. 
Furthermore,  elaborations  were  less  likely  in  later  discussions,  and  students  scoring  high  on  the  negotiation  survey  were  more  likely  to 
negotiate  (evidence  of  the  negotiation  survey’s  validity).  Concerning  the  analysis  of  three-event  sequences,  a  common  three-event  sequence 
in  both  groups  was  uncovered:  sharing  (—2)  — ►  question  (-1)  —*  negotiation  (0)  (Odds  Ratio,  6.52,  p  <  0.05).  However,  we  found  no  sta¬ 
tistically  significant  group  differences  in  three-event  sequences  by  using  this  procedure.  Fig.  8  portrays  SDA  findings. 

In  summary,  the  results  showed  that  students  with  access  to  instructor-based  attention  guidance  functionality  displayed  more  ques¬ 
tioning,  elaborating,  and  negotiating  in  their  messages  compared  to  students  in  the  control  group.  Moreover,  sequential  analysis  results 
show  two  significant  transitional  probabilities  in  relation  to  task-related  messages:  sharing  to  elaborating  and  questioning  to  negotiating. 
The  SDA  results  are  consistent  and  also  show  a  common  three-event  sequence  in  both  groups:  sharing  (—2)  — >  question  (-1)  — »  negotiation 
(0).  Taken  together,  these  results  only  partially  support  Hlb. 


Fig.  7.  Transitional  state  diagram  for  control  group  *p  <  0.05;  **p  <  0.01. 1 


e:  transitional  probabilities  less  than  0.1  are  omitted. 


£.  Eryilmaz  et  al.  /  Computers  &  I 


71  (2014)  303-321 


315 


Table  4 

Descriptive  statistics  of  two-event  sequences  in  threaded  discussions. 


Condition  Two-event  sequences 

Sharing  -»  elaborating  Questioning  -»  negotiating 

n  M  SD  n  M  SD 


Instructor-based  attention  guidance  functionality  64  0.38  0.74  49  0.43  0.73 

Control  software  system  75  0.13  0.38  37  0.08  0.28 


6.3.  Switching  from  teacher-based  to  peer-oriented  attention  guidance 

H2  predicted  that  students  who  switched  from  the  instructor-based  to  peer-oriented  attention  guidance  functionality  would  continue  to 
carry  out  high  quality  social  interaction  patterns  focusing  on  challenging  concepts  from  complex  instructional  materials.  Table  7  shows  the 
differences  in  students’  selection  behaviors  during  the  last  discussion  topic. 

In  the  last  discussion  topic,  treatment  group’s  frequency  (11 )  and  proportion  (69%)  of  annotations  focusing  on  challenging  concepts  using 
peer-oriented  attention  guidance  functionality  did  not  differ  significantly  from  their  means  over  the  first  five  discussions  (10.4  and  73%), 
z  =  0.32,  p  =  0.75,  showing  support  for  the  fading  hypothesis.  As  before,  the  treatment  group  had  a  larger  frequency  and  greater  proportion 
of  annotations  focusing  on  challenging  concepts  than  the  control  group  (6  and  29%,  z  =  2.42,  p  —  0.02).  Building  on  Tables  2  and  7,  Fig.  9 
demonstrates  consistent  patterns  between  the  five  earlier  discussions  and  the  sixth  discussion,  showing  that  the  fading  of  the  in¬ 
structor’s  guidance  did  not  sharply  reduce  the  treatment  group’s  annotations  focusing  on  challenging  concepts  from  complex  instructional 
materials. 

Fig.  10  portrays  a  heat  map  derived  from  nine  page  views  by  nine  different  students  in  the  treatment  group  when  they  switched  from  the 
instructor-based  to  peer-oriented  attention  guidance  functionality.  The  heat  map  for  peer-oriented  attention  guidance  functionality  reveals 
a  concentrated  attention  around  a  sticky  message  on  bigger  font  size  text.  This  heat  map  suggests  that  the  key  idea  posted  by  a  student  for 
annotating  this  passage  reflects  a  shared  misconception  by  eight  other  students  who  invest  an  effort  into  developing  a  better  understanding 
of  the  passage,  as  indicated  by  the  red  color  (in  the  web  version). 

Two  coders  independently  coded  a  total  of  85  messages  from  the  last  discussion  topic.  The  Krippendorffs  alpha  (2004)  inter-rater 
reliability  coefficient  was  0.81,  showing  satisfactory  reliability.  All  disagreements  between  coders  were  resolved  by  discussion.  Table  8 
displays  coding  results  at  a  discussion  level  according  to  the  variables  of  interest. 

Statistical  tests  showed  that  none  of  the  frequencies  or  proportions  for  any  type  of  message  or  message  totals  differed  significantly 
between  instructor-based  and  peer-oriented  attention  guidance  functionalities,  showing  that  fading  instructor’s  support  did  not  signifi¬ 
cantly  reduce  the  treatment  group’s  task-related  messages  and  supporting  H2.  In  the  sixth  discussion,  there  was  no  statistically  significant 
group  difference  in  task-related  message  categories.  Moreover,  there  was  no  statistically  significant  difference  between  these  two  groups  for 
their  two-  and  three-event  message  sequences. 

6.4.  Control  variables  findings 

The  Krippendorffs  alpha  inter- rater  reliability  measure  for  the  coding  of  the  prior  knowledge  test  was  0.75,  which  indicates  satisfactory 
agreement  beyond  chance.  Table  9  presents  the  descriptive  statistics  of  the  prior  knowledge  test.  There  was  no  statistically  significant 
difference  between  the  treatment  and  control  groups  with  regard  to  the  prior  knowledge  test,  t(22)  =  0.19,  p  -  0.85,  d  =  0.26. 

Table  10  shows  the  descriptive  statistics  and  independent  t-test  results  for  task  complexity  of  six  consecutive  discussion  topics.  Based  on 
Table  10,  there  was  no  statistically  significant  difference  between  the  treatment  and  control  groups  with  regard  to  task  complexity  in  six 
consecutive  discussion  topics. 

7.  Discussion 


With  the  increasing  use  of  CSCL  systems,  it  is  timely  and  important  to  understand  how  to  engage  students  in  productive  collaborations 
with  their  classmates  (Suthers,  2006).  Therefore,  this  study  examined  the  design  and  evaluation  of  two  different  attention  guidance 


^-elaborating  sequence. 


Student  Annotated  te 


MessagelD  Author  Content 


The  "only"  relationship  between  epistemology  and  pedagogy  74 

is  based  not  upon  the  translation  or  mapping  of  an  epistemology 
(on)  to  a  pedagogy,  but  rather  the  selection  of  a  fitting  pedagogy 
to  “teach"  (i.e„  help  the  learner  acquire)  the  epistemology. 

In  other  words,  the  choice  of  a  pedagogy  can  and  possibly 
must  be  "informed"  by  the  epistemology  that  the  learner  should 
acquire,  but  is  not  the  same  as  making  use  of  that  epistemology 
as  a  pedagogy.  75 

Any  teacher  needs  to  have  the  subject  matter 
knowledge  in  order  to  perform  his  or  her  job. 

The  next  step  is  to  transfer  this  knowledge  to  students. 
I  would  add  here  that  even  when  teachers  have 
pedagogy,  it  still  does  not  fully  prepare  them  for 
the  "real"  challenges/ situations  of  diverse  populations 
because  no  one  pedagogy  is  a  one-size-fits-all. 


Student  7  /  think  the  author’s  point  is  that  epistemology 

informs  pedagogy  in  the  same  way  that  standards 
and  knowledge  drive  instruction.  “The  epistemology 
that  the  learners  must  acquire"  implies  a  body  of 
knowledge  to  me  that  must  be  pondered  by  a 
teacher  prior  to  designing  instruction  to  transmit 
what  must  be  learned. 

Student  12  Yes,  epistemology  is  the  subject  matter  knowledge. 


Elaborating 
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Table  6 

Extraction  of  an  actual  discussion  sample  representing  questioning-negotiating  sequence. 


MessagelD  Author  Content 


Student  Annotated  text 
The  goal  of  the  process-product  research  on  instruction  was  to  89 
identify  the  instructional  procedures  that  were  used  by  those 
teachers  whose  students  made  the  highest  gains  on  standardized 
tests  or  tests  developed  by  the  research  team  and  to  compare 
these  instructional  procedures  with  the  procedures  used  by  those  90 
teachers  whose  students  made  the  smallest  gains  on  the  same 


Student  1  I  do  not  have  clear  understanding  of  “process-product"  Does  it 
mean  that  if  a  prescribed  procedure  (a  process)  is  followed, 
the  result  (product)  will  be  the  same?  Is  this  a  cookbook 
approach  to  student  achievement? 

Student  8  I  am  also  having  hard  time  with  this.  My  take  is  that  depending 
on  the  content  the  students,  and  the  context  as  the  instructor 
I  choose  what  seems  to  be  the  best.  For  me,  explicit  instruction  does 
fit  at  times.  Inquiry  and  constructivist  methods  also  find  a  place. 

It  really  depends  on  the  learning  goal.  ..but  I  guess  if  I’m  the  one 
deciding  then  it  really  isn't  constructivist  at  all,  is  it? 


Negotiating 


functionalities  in  an  open  source  anchored  discussion  system.  In  this  section,  we  discuss  why  the  proposed  functionalities  led  to  three 
important  results  and  then  tie  each  result  to  the  theoretical  framework. 

First  and  foremost,  the  results  showed  that  instructor-based  attention  guidance  functionality  in  an  anchored  discussion  system  facili¬ 
tated  students’  selection  of  challenging  concepts  from  complex  instructional  materials.  Concerning  software  design,  this  result  emphasizes 
the  strong  positive  effect  of  font  size  on  capturing  students’  attention  and  is  in  accordance  with  tag  cloud  research  (e.g.,  Bateman  et  al.,  2008; 
Halvey  &  Keane,  2007;  Lohmann  et  al.,  2009;  Rivadeneira  et  al.,  2007)  that  demonstrates  how  tags  with  large  font  sizes  stand  out  to  viewers. 
Complementing  this  result,  the  qualitative  analysis  of  heat  maps  suggests  that  instructor’s  suggestions  encouraged  students  with  low  prior 
domain  knowledge  to  express  their  misconceptions  of  difficult  passages  via  sticky  messages  on  text  and  these  sticky  messages  generated  a 
concentrated  group  effort  to  develop  a  better  understanding  of  those  passages.  To  our  knowledge,  the  heat  maps  presented  here  provide 
initial  insights  into  students’  attention  allocations  and  transformation  during  online  discussions.  These  findings  offer  a  natural  solution  to 
the  critical  problem  of  finding  a  balance  between  offering  students  an  inherently  open  learning  environment  where  they  can  express  their 
own  ideas  and  steering  them  toward  discussing  challenging  concepts  from  complex  instructional  materials  (Kirschner  et  al.,  2006). 
Furthermore,  these  findings  go  beyond  providing  students  annotated  instructional  materials  (Wolfe,  2008)  because  students  still  have  to 
distill  the  key  ideas  on  their  own  when  annotating  difficult  passages.  In  light  of  these  findings,  we  argue  that  the  utility  of  proposed 
instructor-based  attention  guidance  functionality  lies  in  the  continuous  assessment  of  students’  attention  allocations,  its  interpretation 
(diagnosis),  and  the  calibration  of  the  guidance  when  needed.  With  respect  to  our  theoretical  framework,  these  findings  confirm  that 
students  with  low  prior  domain  knowledge  need  guidance  (Hmelo-Silver  et  al„  2007;  Kirschner  et  al.,  2006)  allowing  them  to  focus  on  areas 
where  they  struggle  to  understand  (Renkl  &  Atkinson,  2007).  A  possible  explanation  for  our  findings  is  that  most  of  the  information  seemed 
new  to  students  with  low  prior  domain  knowledge  and  the  visual  cues  reduced  students’  search  activity.  As  shown  by  the  heat  maps  (Figs.  4 
and  5),  the  control  group  students  searched  for  central  domain  concepts  and  principles  (yellow  and  green,  in  the  web  version,  regions  in 
Fig.  5),  while  the  treatment  group  students  were  trying  to  understand  the  conveyed  messages  in  cued  areas  (red,  in  the  web  version,  regions 
in  Fig.  4). 

Second,  our  results  indicate  that  instructor-based  attention  guidance  functionality  in  an  anchored  discussion  system  increased  the 
quantity  of  questioning,  elaborating,  and  negotiating  messages  and  improved  two  transitional  probabilities  in  relation  to  these  messages: 
sharing  to  elaborating  and  questioning  to  negotiating.  Importantly,  the  statistical  discourse  analysis  results  corroborated  that  the  treatment 
group  was  more  likely  than  the  control  group  to  elaborate,  showing  that  the  result  at  hand  is  especially  robust.  These  results  suggest  a 
solution  to  the  problem  of  collaborative  discovery  of  new  knowledge  that  one  in  discussion  yet  possesses  (Peters  &  Hewitt,  2010).  Partic¬ 
ularly,  the  two-event  sequences  identified  in  Table  4  are  similar  to  what  Mercer  (2000)  calls  exploratory  conversation:  students  use  each 
other’s  reasoning  to  modify  their  own  thinking  and  at  the  same  time  they  collaboratively  build  new  ideas.  In  contrast,  the  transition 
probabilities  depicted  in  Fig.  8  are  similar  to  what  Mercer  (2000)  calls  accumulative  conversation:  students  accept  each  other’s  ideas 
without  negotiation  and  they  construct  shared  knowledge  by  the  accumulation  of  already  known  concepts.  On  the  theoretical  side,  these 
results  accentuate  that  effective  learning  is  more  likely  to  occur  if  cues  help  students  spend  less  time  for  searching  central  domain  concepts 


Group  Background  2  messages  ago  Previous  message  Dependent 

Variable 


Fig.  8.  Statistical  discourse  analysis  findings  *p  <  0.05,  **p  <  0.01,  ”*p  <  0.001. 


£.  Eryilmaz  et  al.  /  Computers  &  I 


71  (2014)  303-321 


317 


Table  7 

Selection  ratio  of  task-relevant  information  from  the  last  discussion  topic. 


Condition  Treatment  group 

Instructor-based  attention 
guidance  functionality 
Mean  of  1  st  5  topics 

Frequency  of  student  annotations  focusing  10.4 

on  challenging  concepts 

Total  student  annotations  14.2 

Selection  ratio  of  challenging  concepts  73% 


Control  group 


6th  Topic 


21 

29% 


Fig.  9.  Continuous  trends  in  students'  selection  behaviors. 


iao% 


Fig.  10. 


map  for  peer-oriented  attention  guidance  functionality. 
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Table  8 

Content  analyses  of  first  five  discussions  and  the  last  discussion  topic. 

Messages  Attention  guidance  functionality  Control  software  system 


Task-related  Sharing  13.2 

Questioning  10.4 

Elaborating  6.0 

Negotiating  4.6 

Producing  0.6 

Total  task-related  34.8 

Non  task-related  8.6 

Total  43.4 


Peer-oriented 


Proportion  Frequency  Proportion 

0.30  15  0.32 

0.24  13  0.28 

0.14  5  0.11 

0.11  6  0.13 

0.01  0  0.00 

0.80  39  0.84 

0.20  8  0.17 

1.00  47  1.00 


Frequency  Proportion 


13  0.34 

6  0.16 

1  0.03 

2  0.05 


22 


0.58 


16  0.42 


Descriptive  statistics  and  t-test 


results  of  the  prior  knowledge  test. 


M  SD  95%  Cl 


Treatment  group  12  5.25  0.62  [4.90,5.60] 

Control  group  12  5.08  0.67  [4.70,5.46] 


Note.  Cl  =  confidence  interval. 


and  principles,  which  gives  them  more  time  to  diagnose  and  revise  challenging  misconceptions  (Renkl  &  Atkinson,  2007).  Moreover,  in  line 
with  Suthers  et  al.  (2010)  and  Zemel  et  al.  (2007),  the  statistical  discourse  analysis  results  showed  that  a  message  in  a  threaded  discussion 
contributing  to  such  an  understanding  can  depend  on  a  cumulative  series  of  immediately  preceding  messages,  specifically  this  common 
sequence:  sharing  (-2)  ->  question  (-1)  ->  negotiation  (0).  Taken  together,  the  important  results  at  hand  support  the  value  of  instructor- 
based  attention  guidance  functionality  in  an  open  learning  environment. 

Third,  this  study  found  that  after  the  treatment  group  students  switched  from  an  instructor-based  to  peer-oriented  attention  guidance 
functionality,  they  maintained  their  attention  on  challenging  concepts  from  complex  instructional  materials,  supporting  the  view  that 
fading  the  system’s  support  and  shifting  responsibility  to  students  does  not  hinder  these  students’  performance  on  this  task.  This  finding 
extends  the  existing  literature  by  showing  that  building  scaffolding  into  software  allows  students  to  practice  their  knowledge  in  novel 
situations.  Thus,  we  argue  that  building  software  tools  that  eventually  fade  support  offers  a  natural  solution  to  the  fundamental  problem  of 
adaptive  web-based  learning  system  design  (for  an  overview  see  Fischer,  2001).  Moreover,  within  the  scaffolding  paradigm,  peer-oriented 
attention  guidance  functionality  highlights  the  fact  that  social  software  enables  implicit  learning  processes  in  such  a  way  that  an  individual 
student  can  benefit  from  the  collective  knowledge  that  is  inherent  in  a  collaboratively  generated  artifact  (Kimmerle,  Cress,  &  Held,  2010). 
Consistent  with  our  theoretical  perspective,  the  present  finding  suggests  that  the  students  in  the  treatment  group  internalized  the  in¬ 
structor’s  assistance,  though  the  results  of  this  small  sample  on  this  small  time  interval  must  be  interpreted  cautiously. 

8.  Resulting  conceptual  framework 

Drawing  upon  the  theoretical  framework  and  empirical  results  presented  in  our  study,  we  will  now  propose  a  conceptual  framework 
portrayed  in  Fig.  11.  At  the  forefront  of  our  conceptualization  lies  the  distinction  between  shallow  and  deeper  processing  of  text  in  open 
learning  environments.  On  the  one  hand,  students  may  stick  to  shallow  processing  of  text  when  they  gravitate  toward  familiar  topics  by 
sharing  existing  experiences  and  opinions  that  relate  to  and  triggered  by  the  topics  in  the  text.  This  form  of  processing  is  generally  easier  for 
students  and  takes  less  effort.  On  the  other  hand,  deeper  processing  or  getting  to  the  “heart”  of  a  complex  instructional  material  takes  a  lot  of 
effort  because  students  have  to  diagnose  and  revise  challenging  misconceptions  by  elaborating  and  negotiating  ideas,  similar  to  assembling 
a  jigsaw  puzzle  where  every  participant  contributes  a  part  of  a  picture  without  comprehending  the  whole  (see  the  discussion  sample 


Table  10 

Descriptive  statistics  and  t- test  results  of  task  complexity. 

Discussion  topic  Treatment  group  Control  group  t  p 

(n  =  12)  (n  =  12) 


M  SD 

Topic  1 :  the  success  or  failure  of  constructivist  instruction  (weeks  2-3)  4.00  0.74 

Topic  2:  taking  guided  learning  theory  to  school  (weeks  4-5)  4.25  0.75 

Topic  3:  epistemology  or  pedagogy,  that  is  the  question  (weeks  6-7)  4.33  0.49 

Topic  4:  the  empirical  support  for  direct  instruction  (weeks  8-9)  4.5  0.52 

Topic  5:  from  behaviorism  to  constructivism  (weeks  10-11)  3.92  0.79 

Topic  6:  beyond  the  fringe:  building  and  evaluating  scientific  knowledge  4.67  0.65 

systems  (weeks  12-13) 


M  SD 


4.33  0.77 

4.42  0.51 

4.5  0.52 

4.58  0.51 

4.00  0.74 

4.5  0.52 


0.31 

0.19 

0.10 


0.20 


0.76 


0.92 

0.91 

0.94 

0.84 


Note.  Degree  of  freedom  =  22.  Task  complexity  survey  utilized  a  six-point  scale  (1  =  i 


:  all;  2  =  just  a  little  bit;  3  =  somewhat;  4  =  pretty  much;  5  =  very;  6  =  extremely). 
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Fig.  11.  Resulting  conceptual  framework. 


provided  in  Table  6).  Based  on  this  distinction,  we  present  shallow  and  deeper  processing  of  text  as  nested  levels  in  online  social  interaction. 
During  a  discussion,  students  may  go  from  shallow  to  deep  processing  and  vice  versa.  However,  we  have  seen  that  often  only  a  smaller 
portion  of  the  total  discussion  focuses  on  deeper  processing  of  text.  It  is  this  deeper  processing,  where  students  take  on  the  hard  task  of 
struggling  with  a  text’s  meaning  and  advance  their  understanding  of  it.  In  other  words,  the  problem  at  hand  is  that  in  the  educational 
practice  of  collaborative  online  literature  processing  (with  low  prior  domain  knowledge  and  high  task  complexity)  students  often  do  not 
engage  in  this  deeper  level  processing.  Our  results  demonstrate  that  the  proposed  attention  guidance  functionalities  can  afford  an  indirect 
focusing  effect  to  students’  deep  processing  of  the  text  at  hand. 


9.  Conclusion 

The  study  reported  in  this  article  is  part  of  an  action  design  cycle  in  which  we  investigate  the  affordances  and  constraints  of  funda¬ 
mentally  social  technologies  for  mediating  collaborative  knowledge  construction  and  individual  learning  outcomes.  As  a  third  step  in  this 
endeavor,  we  provided  valuable  insights  to  researchers  and  teachers  about  the  trajectory  of  transforming  classrooms  toward  more 
collaborative  knowledge  construction  environments.  In  sum,  an  overall  principle  drawn  from  this  study  is  that  instructor-based  and  peer- 
oriented  attention  guidance  functionalities  in  an  anchored  discussion  system  prompt  students  to  mindfully  interact  with  or  reflect  upon  the 
central  aspects  of  complex  instructional  materials.  Without  guidance,  regular  online  discussion  appeared  to  be  scattered,  inefficient,  and  less 
focused  on  challenging  concepts  from  complex  instructional  materials.  This  type  of  discussion  is  especially  problematic  when  research  on 
the  effects  of  collaborative  learning  is  highly  inconclusive  (see  Kirschner  et  al.,  2009;  Phielix  et  al„  2010  for  reviews)  because  when  mis¬ 
conceptions  remain  undiscovered  students  fail  to  grasp  important  information  (Engelmann  &  Hesse,  2011).  In  contrast,  guided  online 
discussion  was  more  integrated,  efficient,  and  focused  on  intellectual  advances  in  areas  where  students  struggle  to  understand.  Two  major 
implications  can  be  derived  from  the  findings  of  this  study.  First,  students  with  low  prior  domain  knowledge  can  benefit  from  guidance  to 
carry  out  complex  learning-tasks  and  learn  from  them.  Second,  building  scaffolding  into  software  can  help  students  with  low  prior  domain 
knowledge  develop  expertise.  Tools  such  as  the  ones  described  in  this  article  open  a  promising  new  perspective  for  designing  and  using 
software  to  scaffold  learning. 

As  with  all  research,  this  study  has  a  number  of  limitations.  First,  the  present  study  had  a  small  sample  size,  in  part  because  content 
analysis  and  sequential  analysis  are  labor  intensive  and  time  consuming.  A  larger  sample  size  can  increase  the  validity  and  generalizability  of 
our  findings.  Second,  this  study  focused  on  student’s  collaborative  knowledge  construction  process.  We  did  not  examine  individual  learning 
outcomes  developed  from  collaborative  knowledge  construction.  Still,  other  studies  have  linked  deep  processing  of  instructional  materials’ 
central  aspects  with  individual  learning  outcomes  (see  Eryilmaz,  Van  der  Pol,  et  al.  2013  for  an  association  between  knowledge  construction 
activities  and  individual  learning  outcomes).  Third,  it  is  unclear  whether  or  not  participants  in  the  peer  guidance  condition  would  have 
maintained  high  quality  discourse  if  the  condition  persisted  over  a  series  of  discussions.  Thus,  more  research  is  needed  to  assess  the  effects 
of  peer-oriented  attention  guidance  functionality  over  a  long  duration. 

Many  critical  questions  remain  to  be  addressed  under  this  theoretical  framework  as  we  design  more  advanced  tools  to  scaffold  students 
during  online  learning  conversations.  What  are  the  relationships  among  different  types  of  technology-enhanced  scaffolds  and  how  can  we 
fade  them  to  facilitate  adaptive  web-based  systems?  If  students  become  dependent  on  technology-enhanced  scaffolds,  do  they  interact  less 
with  peers  and  instructors?  We  will  examine  these  issues  in  our  future  studies. 
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